Search for: All records

Logic modeling, the process that explicates how programs are constructed and theorized to bring about change, is considered to be standard evaluation practice. However, logic modeling is often experienced as a transactional, jargon-laden, discrete task undertaken to produce a document to comply with the expectations of an external entity, the consequences of which have minimal or even negative influence on the quality of program evaluation. This article presents the Logic Modeling Theory of Action Framework (LMTAF) which elucidates needs, resources, and central activities of logic modeling, and describes its potential evaluation-related benefits. The LMTAF situates evaluators as the primary intended users of logic modeling, and logic modeling as a fundamental element of each stage of a program evaluation life cycle. We aim to reassert the value of logic modeling for evaluation and provide evaluation practitioners a useful touchstone for reflective practice and future action. 

This paper details the process of developing and adapting a narrative framework for teaching an introductory geotechnical engineering course (EGR 340) through a systematic iterative procedure that embeds conceptual learning into a story format and, over time, elaborates elements and interactions within the story using methods of transmedia storytelling. Although the tools are presented within the context of geotechnical engineering, the approach can be applied throughout engineering education. The elaborative transmedia storytelling process we describe is based on the Imaginative Education (IE) teaching approach. Well-grounded in the learning sciences--but novel in engineering education--IE facilitates student engagement through the use of cognitive tools (such as extremes of reality, heroism, and the exploration of binaries). These tools are connected to types of understanding and serve to enhance a sense of mystery and wonder for topics that might not otherwise register as being immediately relevant to students. A significant benefit of this approach is that that it lends itself to modification and personalization through the inclusion of new features and methods of interaction at the level of the whole story and at the level of story elements. Four types of understanding and their associated cognitive tools were used in EGR 340 and their application is described in this paper. They include: • Mythic understanding using a fantasy narrative that played on the idea that geotechnical engineers refer to their field as the “dark arts of engineering.” • Romantic understanding using heroic narratives that helped students put themselves in the place of Terzaghi and Casagrande as they developed the field. Extremes of reality was another Romantic tool used throughout the course. For example, students learned about soil stress by first solving the mystery of how quicksand works. • Theoretic understanding using concept maps and narrative was used at both the course and unit level to organize concepts. • Ironic understanding using discussion of the inadequacies of theoretic understanding to recognize the reference to “dark arts.” Transmedia storytelling through extensive use of short video clips and other means was used to enhance the application of these tools. For example, students traveled virtually to Venice where they joined a noisy gondola tour to examine building foundations and learn about how poor water policies impacted the sinking of the city. Course evaluation and lesson assessment data were collected in 2018, 2020, and 2022, with each year being associated with a different version of the course. Using these data, we present a mixed-methods analysis of learning outcomes that provides evidence for the effectiveness of this approach at different steps along the way. Non-parametric comparisons of student assessment data demonstrated that student conceptual learning was relatively stable across measures and versions, but that students in the fully transmedia iteration generally performed more strongly on assessments of project-based learning (Borrow/Fill; Atterberg; Dam). Thematic analysis of student responses to open-ended course evaluation prompts demonstrates that engagement was high across all versions of the course, and that students in the 2022 version discussed engineering topics in a manner that included personal connections and reflections. 

This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 

This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non- treatment (n = 246) groups, wherein the former participated in a two-to-three- week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann- Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 

This paper examines the use of Imaginative Education (IE) to create an NGSS-aligned middle school engineering curriculum that supports transfer and the development of STEM identity. In IE, cognitive tools—such as developmentally appropriate narratives, mysteries and fantasies— are used to design learning environments that both engage learners and help them organize knowledge productively. We have combined IE with transmedia storytelling to develop two multi-week engineering units and six shorter engineering lessons. An overview of the curriculum developed to date and a more detailed description of the engineering design unit is presented in this paper. The curriculum is currently being implemented in treatment and non-treatment classrooms in middle schools throughout the Springfield, MA public school system (SPS). In tandem with pilot-year implementation of the curriculum, we have developed an assessment instrument to measure student learning outcomes associated with a transfer variant known as preparation for future learning (PFL). An analysis of the results from the PFL assessment support the position that a curriculum employing IE cognitive tools can facilitate both transfer-in thinking and the capacity of students to “think with” and thereby interpret important engineering concepts. 

This paper examines the use of Imaginative Education (IE) to create an NGSS-aligned middle school engineering curriculum that supports transfer and the development of STEM identity. In IE, cognitive tools—such as developmentally appropriate narratives, mysteries and fantasies— are used to design learning environments that both engage learners and help them organize knowledge productively. We have combined IE with transmedia storytelling to develop two multi-week engineering units and six shorter engineering lessons. An overview of the curriculum developed to date and a more detailed description of the engineering design unit is presented in this paper. The curriculum is currently being implemented in treatment and non-treatment classrooms in middle schools throughout the Springfield, MA public school system (SPS). In tandem with pilot-year implementation of the curriculum, we have developed an assessment instrument to measure student learning outcomes associated with a transfer variant known as preparation for future learning (PFL). An analysis of the results from the PFL assessment support the position that a curriculum employing IE cognitive tools can facilitate both transfer-in thinking and the capacity of students to “think with” and thereby interpret important engineering concepts.